Table of Contents

A/B testing

Research purpose

Evaluate the results of the A/B test.

Technical specification

Research description

ab_project_marketing_events.csv — calendar of marketing events for 2020.

final_ab_new_users.csv — users registered from December 7 to December 21, 2020.

final_ab_events.csv — actions of new users in the period from December 7, 2020 to January 4, 2021.

final_ab_participants.csv — table of test participants.

Data preprocessing

Dataset mevents

Dataset new_users

Dataset events

Dataset participants

Interim summary

All names are correct.
There are no duplicates in the data.
There are omissions only in events['details'] - 377577. The details field is filled in only for the "purchase" event.

Test correctness check

Verification of data compliance with the requirements of the terms of reference

Overlap check of the test and marketing events

Test audience check

Growth of indicators check

Expected effect: in 14 days from the moment of registration, users will show an improvement in each metric by at least 10%:

The conversion to viewing product cards product_page fell by more than 10% in 14 days of lifetime.
As can be seen from the graphs, the conversion in the product_page of control group A is 65%, and in the test group 56%

Views of the basket product_cart increased by 3%: group A - 46%, and group B - 49%

Purchases — purchase fell by 6%: group A - 106%, and group B - 100%

Interim summary

The test does not comply with the specification on the following points:

Expected effect: in 14 days from the moment of registration, users will show an improvement in each metric by at least 10%:

The tests overlap with the "Christmas&New Year Promo" marketing events in EU, N.America.

The crossover between the two competing tests was 1,602 users.
There were no intersections of users between the groups of the "recommender_system_test" test.
The groups of the "recommender_system_test" test are not equal to each other.

Data analysis

Number of events per user distribution

Null hypothesis
H0: There are no differences in the number of events per user between the groups.
Alternative hypothesis
H1: There are differences in the number of events per user between groups.

P-value = 0.00000000000000000037 is less than 0.05. We do not reject the alternative hypothesis that there are statistically significant differences in the number of events per user between groups.
The relative gain of Group A of 17% is statistically significant.

Number of events in the samples by day distribution

A/B testing results

Comparison of funnels shows that the conversion rate in test group B has worsened at all stages.
This may be due to the crossover with the holidays, the incompleteness of the test - the set of users was completed later by 2 days and the test was stopped earlier by 5 days, which ultimately excludes most of the users from consideration.

Statistical difference z-test

Null hypothesis
H0: There are no differences in conversion to the target event between the groups.
Alternative hypothesis
H1: There are differences in conversion to the target event between the groups.

Project summary

Comparison of funnels shows that the conversion in test group B has worsened in two stages. This may be due to the intersection with the holidays, the incompleteness of the test - the set of users was completed later by 2 days and the test was stopped earlier by 5 days, which ultimately excludes most of the users from consideration. Also on December 13, there was a sharp jump in the number of events in Group A.

According to the results of the test, there are significant differences between the conversion of groups to "product_page" and "purchase".

There are also significant differences in the number of events per user between groups - the relative gain of group A is 17%.

These results should not be guided, since the test was conducted incorrectly, most of the points of the TOR are not observed.

It is recommended to conduct the test again in compliance with the terms of the TOR, outside of festive promotions.